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ABSTRACT 

Novice instructional design (NID) is an instructional strategy where students design 
instruction in order to better learn information. Novice instructional design (NID) by 
students is an increasingly common instructional activity, particularly with technology; 
however, there is little research regarding what the instructional designer learns by 
engaging in the instructional design process. This paper reports results from an 
experiment investigating what a person learns from novice instructional design of text- 
based content. The experimental design encompasses three independent variables: 
strategy (self-selected learning, novice instructional design); passage structure (well- 
defined, ill-defined); and, achievement (low, high). Dependent variables include concept 
map comprehensiveness, example generation, and task engagement. Several 
statistically significant and theoretically intriguing interactions are interpreted in part by 
Allen & Merrill’s (1985) locus of processing theory for instructional strategies. Together 
with interpreting these interactions, the paper provides both a conceptual background 
and also a research methodology for evaluating novice instructional design of text- 
based content. 
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“The best way to learn something is to teach it” 

Introduction 

Learning through designing is encouraged by a new classroom philosophy where 
students act as researchers, teachers, and monitors of progress as opposed to being 
passive recipients of incoming information (Brown, Ash, Rutherford, Nakagawa, 

Gordon, & Campione, 1993). Supported by current constructivist views of learning, the 
prediction is that generative learning strategies, where the student interacts with the 
material to generate unique learning aids, are more conducive to learning (Wittrock, 
1990). 

Novice instructional design (NID) as described here is an instructional strategy 
where students design instruction in order to better learn information. With the 
increasing presence of technology in the classroom, more students are participating as 
novice instructional designers. There is much constructivist research that describes 
technologically based environments for student instructional designers (e.g., Dugdale, 
1994; Guzdial, 1995; Harel & Papert, 1991; Kafai & Resnick, 1996; Lehrer, Erickson, & 
Connell, 1994; Resnick & Ocko, 1993; Sargent, Resnick, Martin, & Silverman, 1996). 
Such research liberally applies the role of designer to students: for example, including 
eighth grade students designing hypermedia programs about the Civil War (Lehrer et al, 
1994), children developing instructional video games (Kafai, 1995), fourth graders 
designing Logo programs to teach fractions to third graders (Harel & Papert, 1991), 
students creating knowledge bases through novice knowledge engineering (Lippert & 
Finley, 1988) and students designing textile patterns in a Logo microworld (Gargarian, 
1996). The question considered in this paper follows: Is novice instructional design 
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(NID) a valuable learning process? This paper has two purposes: 1) to discuss several 
statistically significant interactions that result from an experiment investigating effects of 
novice instructional design (NID) on learner achievement and engagement; and, 2) to 
present a research methodology for evaluating NID as an instructional strategy. 

Novice Instructional Design as a Learning Strategy 

When considering NID as a learning strategy, the focus is not on the learner-- the 
person to whom the instructional design is directed-- rather, the focus is on what the 
instructional designer learns as a byproduct of designing instruction. As Kafai (1991, p. 
18) states "In the research and theory on design this aspect of what is learned through 
the design process has been neglected"; instead, the research focus has been on what 
designers do instead of what designers learn. As Rowland (1992) explains, there are 
recollections and opinions but little systematically-gathered evidence regarding the 
nature of instructional designing. There have already been numerous studies (e.g., Goel 
& Pirolli, 1992; Greeno, Korpi, Jackson & Michalchik, 1990; Guindon, 1990; Jeffries, 
Turner, Poison & Atwood, 1981 ; Korpi, Greeno & Jackson, 1991; Perez & Emery, 1995; 
Perez, Johnson & Emery, 1995; Rowland, 1992) implementing verbal protocol analysis 
to evaluate cognitive processes in design tasks. These studies were generally 
qualitative, consisting of evaluating and comparing verbal protocols of the instructional 
design process. Importantly, there has not been an attempt to conduct an experimental 
study to assess what instructional designers learn and/or what are the positive learning 
outcomes of instructional design. Consequently, the experiment described in this paper 
investigates areas that have not been previously investigated. 

Another key feature of the experiment is that it focuses on novice instructional 
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design of text-based content in the domain of the social sciences. Most other studies of 
instructional design use material that is technical in nature: e.g. machines (Rowland, 
1992), software design (Jeffries et al, 1981), algorithm design (Kant, 1985), mechanical 
engineering (Ullman, Dietterich, and Stauffer, 1988), vehicle operation (Korpi, Greeno & 
Jackson, 1991), and trouble-shooting a diesel engine (Perez & Emery, 1995; Perez et 
al, 1995). Since most design problems in the real world do not involve manipulations of 
machines, the implementation of a text-based task is applicable to more domains and 
has important implications for text processing as will be described in the next section. 

Textual Organizational Signals 

According to models of textual processing, the cues influencing the reader’s planning 
must be considered. In other words, the structure of text influences a person’s ability to 
organize its content. When processing text, organizational signals, such as headings, 
overviews, and topical summaries, are used to help the reader discern the topic 
structure of a text. These signals may be verbal features (i.e., headings, subheadings, 
pointer words) or visual features (i.e., highlighting, isolated words) that guide the reader 
through text (Lorch & Lorch, 1986). This implies that the presence of organizational 
signals in a text should lead to a more complete and coherent representation of the 
text’s topic structure than would occur in the absence of signals. In fact, it has been 
shown that signals do produce better memory for text topics and their organization 
(Lorch et al., 1993). As described by Lorch and Lorch (1995), the presence of headings 
in a text leads to better summaries (Brooks, Dansereau, Spurlin, & Holley, 1983), better 
outlines (Brooks et al, 1 993; Dee-Lucas & DiVesta, 1 980), and better memory for the 
main points of a text (Spyridakis & Standal, 1987). 
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What if the text is devoid of organizational signals? While the research shows that 
readers may find signals useful aids, they do not require signals to process information 
about the topic structure of a text (Lorch et al., 1985). Under many conditions, readers 
may be able to construct complete and coherent topic structure representations without 
the aid of organizational signals. For example, if there are few topics and the topics are 
well developed and easily related to each other, readers may not need the help of 
signals in constructing a topic structure representation for the text (Spyridakis & 

Standal, 1987). Yet if the text is complex, signals are more useful. 

While there are numerous variables that influence learning from text, two key 
factors explored in this research are the role of novice instructional design as an 
instructional strategy, and the role of passage structure. A third factor considered is the 
achievement level of the student. Taking all three of these factors into account, the next 
section will develop a conceptual foundation for considering these factors in a more 
complex relationship. 



Level of Achievement 

The role of achievement with strategy use has a cogent history. Salomon (1979) 
suggests that a common pattern in many aptitude-treatment interaction studies can be 
explained by the difference between cultivation of a skill and activation of a skill. Skill 
activation is most effective for high-mastery students because they only need to be 
helped to activate existing skills. Skill activation is ineffective for students with low 
mastery because they cannot exercise non-existent skills. Furthermore, high-mastery 
students may suffer decrements in performance when forced to engage in a learning 
strategy because it interferes with existing and successful skills. Building upon the 




7 



Novice Instructional Design 7 



earlier work by Salomon (1979) and Clark (1982), Allen and Merrill’s (1985) locus-of- 
processing continuum predicts that students with low aptitude for a learning task will 
profit most by directed strategies, whereas high aptitude students will profit most by self- 
selected strategies. Recently, Williams (1996) mentioned that such a strategy-selecting 
system has not been sufficiently tested. This study tests the locus-of-processing 
continuum through a text-based task. 

Predictions 

In this experiment, dependent variables reflect differences in three areas: 1 ) 
cognitive structure; 2) ability to generate examples; and, 3) task engagement. 
Consequently, what follows are predictions for each of these areas. 

Concept Map Comprehensiveness 

Concept maps were chosen to reflect the cognitive structure and knowledge 
organization of the content by the students. While it is predicted that the novice 
instructional design strategy will facilitate development of a more comprehensive 
concept map, it is uncertain as to the interacting roles of strategy, passage structure, 
and achievement. 

In this study, students will brainstorm concepts and then organize them into a 
concept map. This approach is preferable to providing a list of concepts for mapping 
because it provides a picture of subjects’ individual and idiosyncratic views of a content 
domain (Winitsky, Kauchak & Kelly, 1994). Following a constructivist conceptual 
framework, it is also not of interest how the concept maps compare in terms of the 
similarity of a student’s concept map to that of an expert (e.g., Beyerbach, 1 988; 
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Morine-Dershimer, 1991). In contrast, the students’ individualized concept maps are of 
interest. 



Example generation 

An indicator of learning from instructional design is the learner’s ability to generate 
examples in the content area. It is predicted that the novice instructional design strategy 
will facilitate example generation. 

Synthesizing the research literature, Baylor (1997) derived the following 
components of instructional design as a cognitive problem-solving process: 
decomposition, transformation, evaluation, and revision. The first stage of instructional 
design is decomposition, where a person breaks down the initial problem into primary 
components and re-creates the structure of the problem. From Baylor’s (1997) 
delineation of design stages, the second stage following decomposition is 
transformation of the design information into a mental model. Since part of the 
transformation process is generating information, it follows that one could generate 
ideas or examples in this stage. As Lehrer (1993) describes, transforming information 
into knowledge includes such processes as finding information, developing new 
information, selecting information, organizing information, and representing information. 
Consequently, example generation could reflect the extent to which the novice 
instructional designer has transformed the information. 

It is also of interest how passage structure may influence example generation given 
that the information is text-based. As Simon (1973) explained, much problem solving 
effort of design is directed at structuring problems (i.e., the decomposition stage), and 
only a fraction of it at solving problems once they are structured (p. 1 87). If using a well- 
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structured passage eliminates some of the need for the designer to restructure the 
design problem (the text) in the decomposition stage, the designer could focus on the 
next-level design stage of transformation which facilitates example generation. 

Task engagement 

In terms of task engagement, it is predicted that the learner will find instructional 
design to be more engaging than self-selected learning (where the learner learns the 
information in any way s/he chooses). The literature suggests that instructional design is 
supposed to be engaging (e.g., Lawson, 1980; Murphy, 1992) and that by working as 
instructional designers, students demonstrate more motivation and effort as well as 
implement higher order thinking skills (Carver, Lehrer, Connell, and Erickson, 1992). 

The next section will discuss the experimental methodology developed for 
evaluating student novice instructional design of text-based content. 

Research Design 

The research design is a 2 x 2 x 2 factorial, with eight cells and approximately 
fifteen students in each of the eight conditions. The three factors are strategy (NID, self- 
selected learning (SSL)), passage structure (well-defined: operationalized as with 
headings and italicized concepts; ill-defined: without headings and italicized concepts) 
and level of achievement (high, low). 

In the self-selected learning (SSL) strategy, students read the passage and learned 
the content in the passage however they chose. In the novice instructional design 
strategy, students read the passage and designed instruction for content in the 
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passage. In both strategies, students could refer to the passage. Half of the students 
used a well-defined passage while the other half used an ill-defined passage. The main 
purpose of the experiment was to compare how designing instruction for text affects the 
following dependent variables: construction of a concept map; example generation; and, 
task engagement. 

Methods 



Participants 

The sample includes one hundred and twenty-three general psychology students 
from a Southeastern community college that were randomly assigned to a treatment. 
Students represented a variety of fields and background experiences. The mean age 
was 25 years with a standard deviation of 8.4 years. Sixty-five percent of the students 
were female and thirty-five percent were male. Ethnicity was fifty-two percent White, 
forty-two percent African-American, and six percent other ethnic groups. Students from 
general psychology classes shared a similar syllabus, course objectives, and textbook. 
The study was implemented after the midterm of the semester so as to exclude those 
very low-achievers who were likely to drop the class without penalty. Students 
participated in the treatment activities either during their assigned class period or during 
one of three voluntary sessions held at other times. Since students were all from 
general psychology classes, they were prepared to comprehend the content of the 
passage used in this study given that it described developmental gender differences. 

Procedure 



After collecting informed consent forms, the researcher presented students with two 
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booklets, a purple booklet containing a passage on gender differences and a yellow 
booklet containing experimental materials that differed according to treatment group 
(NID or SSL). The passage is approximately 2000 words in length. The passage is 
based on modified text regarding gender differences from a general psychology text 
(Weiten, 1995, pp. 462-467). The passage described actual cognitive and social 
differences, environmental origins of the differences (e.g., socialization), and the status 
of gender roles in transition. 

After providing personal information including social security number, gender, and 

age, students were informed that they would be awarded extra credit points based on 

their overall effort in the study. Following this, students were given ten minutes to read 

the passage through one time. After ten minutes, students turned to page one in the 

experimental booklet. Students in the SSL strategy condition were instructed as follows: 

Referring to the passage, learn the content in the passage so that you can do well on an exam. 
You may write notes below. 

Students in the NID strategy condition were instructed as follows: 

Referring to the passage, plan how you will teach the content in the passage to a classmate. Your 
goal is for your friend to learn the material as well as possible so that s/he can do well on an 
exam. You may write notes below. 

All students were given ten minutes to either learn or prepare instruction, depending 
on their strategy. Both treatments provided incentive for the students to perform well. As 
listed in the instructions, the student anticipated an exam on the content for him or 
herself (in the SSL strategy condition) or anticipated an exam on the content for 
someone s/he would instruct (in the NID strategy condition). 

Following this intervention, students were instructed to close the passage booklet 
and turn the page in the yellow booklet, where they were presented with the following 
two questions to elicit example generation: 
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1 . List ways that you could encourage a male to acquire traditionally feminine characteristics. 

2. List your ideas for characters for a children’s television show who will discourage stereotypical 
gender differences. (For example, an older woman who is a mathematics expert.) 

Students were given ten minutes to answer the two questions on this page. 

After answering these questions, students listened to a lecture on creating concept 

maps and also viewed sample concept maps. Following this instruction, practice, and 

without referring to the passage, students were given ten minutes to draw a concept 

map representing the material that they read in the passage. The instructions for 

creating a concept map were adapted from Winitsky & Kauchak (1995): 

Please do not refer to the passage at this time. Create a concept map below to describe your 
understanding of the content in the passage. There are no ‘correct’ answers. What is important is 
that the map you construct reflects your thinking about gender differences. Please try to make 
your map as complete and as thorough as possible to provide an accurate picture of your thinking 
regarding the content. 

Following ten minutes and the creation of this concept map, students answered 
questions regarding their curiosity about gender differences and task engagement. 

Other data that was collected include achievement data, which consisted of exam 
scores in psychology. This data was collected by social security numbers from 
psychology instructors of the participating students. 

Note that a more detailed description of procedures, scoring, and reliability 
information is included in Baylor (1997). 



Results 

The following three sets of results will be discussed: 1 ) concept map 
comprehensiveness; 2) example generation of high achievers; and, 3) task 
engagement. As mentioned previously, each of these dependent variables was 
examined by a three-factor AN OVA ( strategy : SSL or NID; passage structure : well- 
defined or ill-defined; and, achievement level: high, low). 
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The Complexity of Concept Map Comprehensiveness 

It was predicted that there would be a complex relationship among the three factors 
(strategy, passage structure, achievement level) in influencing concept map 
comprehensiveness. Correspondingly, the results present a complex relationship of the 
factors. 

As mentioned previously, concept maps were chosen to reflect the cognitive 
structure and knowledge organization of the content by the students. The analysis for 
overall concept map comprehensiveness is comprised of three components: the themes 
included, organization, and extensiveness of the concept maps. 

First, the content of the concept maps was qualitatively assessed by determining 
whether three themes from the passage were represented. Second, in terms of logical 
organization, the concept maps were rated on a scale from one to four where four 
indicates a high level of organization. The scoring criteria are described in Baylor 
(1997). Third, concept map extensiveness was determined by counting the number of 
valid concepts on the concept map. 

An overall concept map comprehensiveness score was determined by taking 
themes, organization, and extensiveness into account. The mathematical formula to 
obtain this score weighs the three factors relatively equally: (concept map overall score) 
= (extensiveness) + (organization X 10) + (10 points for each of the three themes that is 
represented). This overall score served as the dependent variable for a three-way 
(strategy X passage structure X achievement) between-subjects analysis of variance. 

Interestingly, there was a 3-way interaction of strategy, passage structure, and 
achievement [F(1,103)=4.27, p<.05], as shown in Table 1. There was also a main effect 
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for achievement which would be expected (i.e., that high achievers score higher than 
low achievers) and is not of interest. The three-way interaction, with the means listed in 
Table 2 and shown graphically in Figure 1 indicates that low achievers develop better 
concept maps in the well-defined passage with the SSL strategy, and in the ill-defined 
passage with the NID strategy. For high achievers, the reverse is true: they developed 
slightly better concept maps in the ill-defined passage with the SSL strategy and in the 
well-defined passage with the NID strategy. But what does this mean ? 

Building upon earlier work by Salomon (1979) and Clark (1982), Allen and Merrill’s 
(1985) locus-of-processing continuum would predict that students with low aptitude for a 
learning task will profit most by directed strategies, whereas high aptitude students will 
profit most by self-selected strategies. With the addition of passage structure as a third 
factor, it provides an interesting twist for interpretation in terms of the locus-of- 
processing continuum. 

First recall that the significance of a three-factor interaction indicates that the simple 
interaction effects of two variables are not the same at different levels of the third 
variable (Keppel, 1991, p.427). In this case, we will consider the effects of passage 
structure and achievement at different levels of strategy (NID, SSL). 

While studying Figure 1 (and after priming your extra working memory stores), first 
examine the SSL strategy where students were directed to apply their own strategy to 
learn the content in well- and ill-defined passages. Interestingly, high achievers have 
nearly equivalent concept map comprehensiveness score for both the well-defined 
(X=60.05) and ill-defined (X=61.93) passages. Low achievers, on the other hand, create 
better concept maps when they were provided with the well-defined passage (X=53.00 
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vs. 47.33). The better performance for the low achievers in the well-defined passage 
may be because the textual organizational cues facilitated their usual organization of 
the text. 

The case of the NIP strategy is an interesting corollary. Here, the high achievers 
create better concept maps (X=61.71 vs. 50.19) in the well-defined passage. Perhaps 
with the text figuratively “prepped” for forming an instructional design organization of the 
material, the high achievers naturally took advantage of the textual cues in 
implementing instructional design as a learning strategy. The low achievers, on the 
other hand, developed better concept maps when given the ill-defined passage (X=53 
vs. 41 .07), indicating that they were not savvy enough to recognize the passage 
structure cues as being relevant to instructional design. 

Moving from viewing a glimpse of the cognitive structure of the students, the next 
section will discuss the performance of the students in terms of example generation. 

Example Generation Trend for High Achievers 

It was predicted that NID would facilitate example generation. However, the results 

paint a more complex picture when focusing on the high achievers. 

Three two-way ANOVAs with strategy (SSL, NID) and passage structure (well- 
defined, ill-defined) as the two factors were executed for all three dependent variables 
involving example generation. While there were no statistically significant results for the 
total group or for the low achievers as a group, there were interesting findings for the 
subset of high achievers. For the higher achievers, all three ANOVAs had significant 
interaction effects of strategy and passage structure: 1) generating socialization 
strategies [F(1,65)=5.80, p<.05]; 2) generating gender-sensitive television characters 
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[F( 1 ,65)=7.80, p<.01]; and, 3) example generation total, a composite of the prior two 
scores, [F(1 ,65)=4.90, p<.05], The means are listed in Table 3 and a sample interaction 
figure is included in Error! Reference source not found.. 

Recall again that these example generation analyses were performed on high 
achievers only (A/=69) and adjustment to alpha was made (alpha=.01 7) for the three 
tests. These results continue to highlight the role of passage structure as it relates to the 
two tasks of SSL and NID. In considering the high achievers, it is assumed that they 
would be more strategic than the low achievers when given a particular combination of 
strategy and passage. Notably, in all three cases of example generation, high achievers 
generated the most examples in the SSL strategy with the ill-defined passage and in the 
NID task with the well-defined passage. This suggests that the structure of the well- 
defined passage facilitated example generation for NID but was detrimental for SSL. 
Perhaps in the self-selected learning strategy, subjects were more likely to generate 
examples if they had to impose a structure on the text (i.e. , through evaluating the ill- 
defined passage). In contrast, for the NID task, example generation was facilitated when 
the student was provided with more textual structure. 

The example generation findings indicate that the well-defined passage was 
beneficial for NID, but detrimental for SSL. One explanation could be that the designer 
“borrows” structure provided by the headings and italicized concepts and then examples 
“fit” this structure which is reinforced through the design process. With respect to the 
cognitive task of summarization, Kintsch (1990) found that college students did better at 
the summarization task when the text was poorly structured, implying that students 
benefit from deeper processing to impose structure on the text. Since instructional 
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design is another cognitive process that requires the student to impose structure on 
material, possibly this finding on summarizing could be extended to include instructional 
design as well. 

As Simon (1973) explained, much problem solving effort of design is directed at 
structuring problems (i.e., the decomposition stage), and only a fraction of it at solving 
problems once they are structured (p. 187). An explanation following this would be that 
using the well-defined passage eliminated some of the need for the designer to 
restructure the design problem (the text) in the decomposition stage; consequently, the 
designer could focus on the design stage of transformation, which is a higher-order 
task. From the perspective of SSL, it may be better for him/her to create a personalized 
structure of the textual information. Or, perhaps the self-selected learner finds the well- 
defined passage structure to be distracting while s/he is trying to learn the textual 
content, and interfering with his/her ability to generate examples. 

The next section will move from evaluating performance to evaluating perceived 
engagement of NID and SSL by the students. 

Low Achievers More Engaged Bv Instructional Design Task 

It was predicted that those using the novice instructional design strategy would 
report the task as more engaging than those in the self-selected learning strategy. 
However, the results indicate to the contrary for the high achievers. 

A three-way (strategy, passage structure, achievement) between-subjects ANOVA 
was performed to determine what factors influence task engagement scores for the 
treatment task. There was a significant interaction of strategy and level of achievement 
(F(1 , 1 1 5)=6.57, p=.01 ). Interestingly, the two-way interaction indicates that low 
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achievers reported more engagement in the NID task and high achievers reported more 
engagement in the SSL task. 

Perhaps the high achievers saw this role of “instructional designer” as distracting 
and tedious in comparison to their preferred ways of learning. In contrast, the “role 
playing” aspect of instructional design may have appealed to the low achievers. In other 
words, perhaps the low achievers were less accustomed to finding success and positive 
affect in school tasks (e.g., Kagan, 1990), and were thereby more intrigued by the 
fantasy, novelty, sociability, and power associated with instructional design. Allen and 
Merrill’s (1985) model would also predict that high achievers would prefer SSL 
strategies because it would facilitate them in choosing their preferred learning strategy. 

Even so, designing is supposed to be overall more engaging (e.g., Lawson, 1980; 
Murphy, 1992) and there was no evidence through a main effect that students preferred 
the design task over the other experimental tasks. There are two reasons why it may 
not have been engaging in this study. First, the literature indicates that one relevant 
factor may be situating the designing task in a technological environment (e.g., Harel, 
1991; Kafai, 1995). In a technological environment the design task is more complex and 
with feedback it may be more motivating. Second, students were not focused upon the 
instructional design product (i.e., artifact) that they created in this study. The literature 
shows that creating an artifact can be a motivating process for the designer as the 
designer addresses an unseen audience (e.g., Guzdial, 1995). 

Conclusions 

The main contribution of this study is in terms of methodological development. 

Since this was the first experimental study to examine learning from text through 
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instructional design, a methodology for textual processing was developed that can be 
built upon for future research in this field. 

Overall, the findings demonstrate the complexity of the role of passage structure 
and achievement with novice instructional design in the following three areas: concept 
map comprehensiveness, example generation, and task engagement. First, concept 
map comprehensiveness, as it represents cognitive structure, indicates that mental 
models are influenced by achievement level and passage structure, together with 
strategy. Second, in terms of example generation, only higher achievers were 
considered. As predicted by Allen & Merrill’s (1985) locus of processing model, the well- 
defined passage hindered example generation in the SSL strategy condition, while 
facilitating it in the NID strategy condition. This was found with all three example 
generation measures. Third, in terms of task engagement, it was found that the low 
achievers preferred the NID task whereas high achievers preferred the SSL task. 

Given that the main task in this study involves instructional design of text, it is 
important to consider the effects of the textual organization, or passage structure. 

Based on textual organization literature, it would have been expected that the well- 
defined passage would facilitate performance in both strategies (NID, SSL). The lack of 
a main effect for passage structure together with the series of interactions with passage 
structure is significant in that it suggests that the role of passage structure is complex. 

Several qualifications for the research results should be considered. First, the lack 
of main effects should be noted. However, it is not surprising that there were no main 
effects -- they were not really expected given the complexity of the situation. In future 
research, these questions may be better explored through multivariate analysis. 




20 



Novice Instructional Design 20 



Second, it is important to note that concept maps may only provide a glimpse of a 
person’s cognitive structure and may not be necessarily representative of a person’s 
mental model. Third, together with limitations in assessing knowledge structures, the 
text-based instructional design task in this study was most likely not complex enough to 
elicit extensive problem solving. Future research could include a richer source of 
material, such as technology or other media, and perhaps creating an artifact. Of 
course, this study highlights the limitations of an experimental approach; while it allows 
for more control it sacrifices in terms of ecological validity. While it may be more 
ecologically valid to use a richer design task, it would be more difficult to study it 
experimentally. Fourth, these findings warrant replication, even with the large sample 
size in this study. 
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Table 1 . ANOVA Summary Table for Overall Concept Map Score 



Source 


df 


SS 


MS 


F 


Strategy 


1 


454.34 


454.34 


1.19 


Passage Structure 


1 


19.38 


19.38 


.05 


Achievement 


1 


2648.80 


2648.80 


6.93** 


Strategy X Passage Structure 


1 


29.94 


29.94 


.08 


Passage Structure X Achievement 


1 


429.92 


429.92 


1.12 


Strategy X Achievement 


1 


24.83 


24.83 


.06 


Strategy X Structure X Achievement 


1 


1633.81 


1633.81 


4.27* 


Error 


111 


42422.11 


382.18 




Total 


118 









*p<05, **p<01 
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Table 2. Mean Concept Map Comprehensiveness Scores 



SSL Strategy NIP Strategy, 



Variable 


N 


Mean 


SD 


N 


Mean 


SD 


Ill-defined passage 
Low achievers 


15 


47.33 


16.18 


15 


53.00 


12.56 


High achievers 


14 


61.93 


24.32 


16 


50.19 


20.02 


Well-defined passage 
Low achievers 


7 


53.00 


25.04 


14 


41.07 


18.28 


High achievers 


21 


60.05 


21.33 


17 


61.71 


18.78 




28 



Novice Instructional Design 28 



Table 3. Mean Example Generation Question 2 Scores for High Achievers 







SSL 






NID 




Variable 


N 


Mean 


SD 


N 


Mean 


SD 


Generate socialization strategies 


Ill-defined passage 


14 


5.43 


2.59 


16 


2.69 


1.74 


Well-defined passage 


22 


4.09 


1.54 


17 


4.00 


3.04 


Generate television characters 


Ill-defined passage 


14 


6.64 


4.22 


16 


4.31 


2.75 


Well-defined passage 


22 


4.91 


4.32 


17 


6.59 


3.14 


Total example generation 


Ill-defined passage 


14 


12.07 


5.69 


16 


7.00 


3.58 


Well-defined passage 


22 


9.00 


4.89 


17 


10.59 


5.22 
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Figure 1 . Three-way Interaction of Strategy, Level of Achievement, and Passage 
Structure for Concept Map Comprehensiveness 
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Figure 2. Example of Three-way Interaction of Strategy, Level of Achievement, 
and Passage Structure for Example Generation Question 2 
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